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Abstract 

We describe a model-based vision system to assist the pilots in landing maneuvers under 
restricted visibility conditions. The system has been designed to analyze image sequences obtained 
Irom a Passive Millimeter Wave (PMMW) imaging system mounted on the aircraft to delineate 
runways/taxiways, buildings, and other objects on or near runways. PMMW sensors have good 
response in a toggy atmosphere; but their spatial resolution is very low. However, additional data 
such as airport model and approximate position and orientation of aircraft are available. We exploit 
these data to guide our model-based system to locate objects in the low resolution image and 
generate warning signals to alert the pilots. We also derive analytical expressions for the accuracy 
ol the camera position estimate obtained by detecting the position of known objects in the image. 


I. Introduction 

Federal regulations specify the minimum visibility eonditions under which airlines may take 
oil and land. These minima are a function of the types of airplane and airport equipment. Therefore, 
there is a great deal ot interest in imaging sensors which can see through fog and produce a real 
world display which, when combined with symbolic or pictorial guidance information, could provide 
the basis lor a landing system with lower visual minimum capability than those presently being used 

Since the energy attenuation in the visible spectrum due to fog is very large 12] (Fig.l), 
sensors are being designed to operate at lower frequencies (e.g. 94 GHz) where the attenuation is 
lower providing the ability to see through fog. NASA Langley Research Center, in cooperation with 
industry, is performing research on an on-board imaging system using a passive sensor operating at 
this frequency. Images from such sensors are of very low spatial resolution (Fig. 2). However, 
additional supporting information in the form of knowledge about the airport and the position, 
orientation and velocity ol aircraft is generally available. Thus a model-based image analysis 
approach is feasible to segment the image and to detect and track objects on the ground. Information 
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extracted from such an analysis is uselul to generate warning signals to the pilot of any potential 
hazards. This paper describes such a model-based technique, which makes use ot a priori 
information about the geometric model of the airport and camera position and attitude data provided 
by the Global Positioning System (GPS) and other instruments. 

The geometric model of the airport contains positions ot the runways/taxiways and buildings, 
(he navigation instruments provide the position ot the aircrati, and on-board instruments provide the 
orientation of the aircraft (yaw, pitch and roll). We use this information to detine regions ot interest 
in the image where important features such as runways/taxiways, the horizon, etc. are likely to be 
present. Edges corresponding to these features of interest are detected within these regions. Alter 
delineating regions representing runway/taxiways. we look for objects inside and outside these 
regions. 

The data from radio navigation instruments are known only upto a certain accuracy depending 
upon the type of radio navigation instruments. For example, GPS data is updated once every second 
and it is likely that a few such updates are missed making camera position data to be a tew hundred 
feet off. On-board instrument data is generally useful to obtain more accurate camera position data 
than the GPS-based data. An alternative approach is to use the information about the location ot 
detected objects in the images with known world coordinates (e.g. intersection ot runways/taxiways, 
corners of buildings, etc.) to obtain an improved estimate of the camera position. This requires an 
analytical study of the relationships among the camera parameters, the resolution ot the images, and 
the distances between the aircraft and objects. 

In Section II we present a block diagram of the complete system. In Section III we describe 
the analytical model that establishes the relationship between the position, orientation and other 
physical parameters of the camera and the attributes of the captured images. This model is useful to 
calculate the accuracy of camera position estimation using image based leatures. In Section IV we 
present the method for defining the regions ot interest in the image using the camera parameters and 
airport model. Section V includes image processing steps that are used to tind regions 
corresponding to major features in the image and to detect objects in these regions. Experimental 
results are presented in Section VI. We conclude the paper with a summary and a briet description 
of future work. 
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Fig. 1. Atmospheric effects on electromagnetic radiation [2]. 
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Fig. 2. The Passive Millimeter Wave image. 


II. System Description 

In this section, we describe the functions of various modules of the system shown in Fig. 3 
and the interactions between them. The input model of the airport contains positions of the 
runways/taxiways. and buildings. The model transformation module will take this model and the 
camera state information (position and orientation) as inputs to define the regions of interest in the 
image plane. 


l |na S c Airport 



GPS On board Instruments 


Fig. 3. System block diagram. 






The image processing algorithms in the feature detection module operates within these 
regions of interest to detect the edges of the runway, horizon, etc. in the image. An edge is fitted to 
the edge pixels if enough edge pixels are found within the region of interest. The module outputs 
parameters which define major regions in the input image. ' 

The object detection module detects objects in the image using different thresholds for each 
region. For example, since detection of objects on the runway is extremely important, a lower 
threshold is used to tlag every object even if the contrast is low whereas a higher threshold is used 
to detect objects which are outside the runway such as buildings, etc. Locations of detected objects 
with known world coordinates is useful to estimate camera slate parameters. 

The motion estimation module uses dynamic scene analysis methods to estimate camera 
state parameters as well as to detect velocities of objects on the ground. The outputs from this 
module will be useful to detect potential collisions and generate warning signals as appropriate. 

The camera state estimation module integrates information obtained about the position and 
velocity of the aircraft from various sensors and modules and outputs necessary data to the model 
transformation module. 


III. Accuracy of Camera State Estimation from Image-based Features 

As we need to use the camera state estimated from locating features of known objects in the 
image during the period when the GPS is not updated, it is necessary to know the accuracy of such 
estimated positions and the factors that decide the accuracy. Hence, an analytical model that 
establishes the relationship between the camera parameters and the attributes of captured images is 
necessary for guiding the image analysis system. Sensor positional parameters include range 
(distance from the aircraft to the runway threshold), cross range (distance from the aircraft to the 
runway center line), altitude, and pitch, roll and yaw angles. Sensor imaging attributes include the 
number of pixels in the image and the optical angular view measured in degrees. We derive the 
inter-relationships among these parameters. Using these relationships we calculate the accuracy of 
the estimate of camera position based on a minimum resolvable movement of features by one pixel in 
the image. We obtain these accuracies for three different types of cameras (PMMW. FLIR. HDTV) 
at six ranges. 

A. Analysis 

Throughout the analysis, for convenience, we assume that the sensor is located at the center 
of gravity of the airplane. Hence we can use the terms sensor position and aircraft position 
interchangeably. We also neglect the effect of curvature of the earth. The system of reference axis 
that forms the basis of system of notations used to describe the position of the sensor is shown in 
Fig. 4. The figure shows an airplane with three mutually perpendicular axes — pitch, roll and yaw — 
passing through the center of gravity of the airplane. The three angular displacements are termed 
pitch, roll and yaw as shown in Fig. 4. The image plane is assumed to be perpendicular to the rolling 
axis with its vertical and horizontal axes coinciding with the yawing and the pitching axis of the 
airplane, respectively. 

Fig. 5 shows an imaging situation during landing where the aircraft is at (X c , Y c , Z c ), with 
pitching angle 0, zero yaw and zero roll angle. Let a = 90 - 0 . The field of view of the camera is 
determined by two viewing angles: Act defined in the same plane as 0and A/3 at right angles to Act 
( Act determines the vertical extent of the image and A/3 its horizontal extent). Even though the 
image obtained by the sensor is always a rectangle, the ground area captured by the sensor is a 
trapezoid A BCD whose side length and area depends on Act, A/3 and various other sensor 
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parameters like position, orientation etc. Note that a pixel in the image plane corresponds to a patch 
on the ground plane. We refer to this as a pixel-patch (see Fig. 6). 

Consider a point feature which has been detected at some pixel ip, q). Let the actual world 
coordinates of this feature be ( P, Q, 0). Since a pixel represents a patch on the ground, the camera 
could change in its position by certain amount while still retaining the image of the feature at the 
same pixel (p, q). Hence a camera pose estimation by passive triangulation will always give the 
same camera pose for nearby camera positions unless the change in camera position is large enough 
for the feature to be observed in the neighboring pixel. We define this minimum change in camera 
displacement as the sensitivity of the camera. Note that this is a measure of accuracy of camera 
position estimate and is a function of the camera, image size in number of pixels, angular resolution, 
and the pixel location ip, q) in the image plane. 

Let N x and N v represent the number of pixels in the vertical and horizontal directions. 

respectively. The pixels are numbered -N x /2 0 N x /2-l in the vertical direction and 

-N x /2 0 Ny/2-J in the horizontal direction. The rolling axis of the plane is assumed to pass 

through the bottom right comer of the patch on the ground plane which corresponds to the center 
pixel in the image plane. Other pixels are referenced in a similar manner. The coordinates ot the 
reference corner of the ground area covered by a pixel ip, q> can be estimated by the tollowing 
relations. 


Acc 

X = X ( . + Z c tan( a + p ) 

^ x 

Z„ . Ap 


Y=Y +■ 


cost a + p ) 

N x 


— — tan(</ ) 

Aa ^ n 


( 1 ) 




World coordinate system v e ri.e=i or yc—nq «.» 

Fig. 4. Airplane-body axis (Reproduced from “Airplane Aerodynamics"’ 
by Dommasch and Danieol Otto |ed. 19671). 
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Fig. 5. Image obtained by the sensor is projected towards the ground. 
Hatched portion is the ground area covered by the sensor. 


For a non zero rolling angle <3. the ground coordinates (X’, Y ) which corresponds to a pixel ip, q) in 
the image plane are obtained by replacing (p, q) in the above equation by (p , q ), where 

p = /?cos0-</sin</>, (2) 

q = p sin0 + q cos 0. 

Since a pixel-patch is referenced by its bottom right comer ot the pixel, the other three comers 
become the reference of its three neighboring pixels-patch as shown in Fig. 7. Thus, the lour comers 
of this pixel-patch, (X,\ Y,’). i=/,2,3,4, are obtained by using Eq. (1), where (p, q) are replaced by 
(p,\ q,'), where 

p, = p t cos0-q,s\n0, ( 3 ) 

q, = /vsin 0 + t/,cos< 2 >. 

and ip,, q,) - <p, q), <p : , q 2 ) = (p+1, q), (pp q<) = (p+1, q+1 )• ^nd (p 4 , q 4 ) = (p, q+1 )■ 

Eq. (1) explicitly gives the relationship between the camera parameters <X C , Y c , Z c ), 9,0, 
and a ground point corresponding to a pixel ip, qj. We are now interested in computing the 
sensitivity of the imagery sensor. This is defined as the minimum change in a camera parameter that 
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would move a lixed ground point to the next pixel in the image plane. We obtain this hy taking the 
partial derivative of A"/’ and K/' with respect to the corresponding parameter. For example. 
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Fig. 6. Ground area covered hy the sensors. 
Each small trapezoid corresponds to 
a pixel in the actual image. 


Fig. 7. A pixel (p, q) projected 
towards the ground. 


This derivation is an approximation to the amount of change in Aj for unit change in X c . Thus 

we estimate that the amount of change in X ( in order to change Aj to AS, or F[ to (which define 
ihe corners of adjacent pixels) as 


Sx,, = — =Zy — • and Sy . = — 3 
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Note that = 00 . as expected. Sensitivity with reference to other parameter is defined in a similar 


manner. These are summarized in Table I. 

Sensor sensitivity is a function of various sensor parameters and sensor attitudes. Since the 
sensor plane is inclined to the ground plane, the sensitivity varies in the vertical and horizontal 
direction along the sensor plane and hence is a function of pixel number (p, q). Equivalently, the 
accuracy ol estimation ot sensor position using ground truth data is a function of pixel position as 
well as other parameters. For a given range, the estimation using features that are observed at the 
top hall ot the sensor are less accurate because of the large ground area represented by these pixels. 
Also tor a given p, the accuracy decreases as we move towards the border of the sensor in the 
horizontal direction. In summary, the accuracy of estimation is a function of sensor characteristic and 
the ratio of the sensor view angle to the number of pixels in the image. 
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Sensor Characteristics 
Vertical Horizontal 

Field of view A a A (3 

Number ol pixels N x Ny 

Sensitivity: Minimum chance in the sensor positional parameters (X c . Y c , Z c , 0, 0) that will make the object to 
appear in the next pixel either in the vertical (X; hence called as sensitivity in x direction) or in the horizontal (Y; 

hence called as sensitivity in v direction) direction ol the sensor plane. 5/ : Sensitivity in the direction T due to the 
sensor positional parameter Y computed at pixel (p, q) in the image plane. 


SI 1 ! 1 : Sensor Positional Parameters 

(X c . Y c , Z c ) Sensor position 

0 Pitch angle 

0 Roll angle 


Table I. Sensor positional sensitivity equations. 
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B. Quantitative Results and Discussions 

The sensitivity analysis described in the previous section was applied to three dilterent 

sensors at six different positions (Table ID. Sensitivities Sg c , Sy c and Sz c • al l ^e aim point (i.e.. 

y 

p=0. q-0) for various sensor positions are plotted in Figs. 8, 9 and 10 respectively. Note that S^ c bs 
larger than Sy at (0, 0) and hence a feature would move to the next horizontal pixel before it moves 

y 

to the next vertical pixel. Thus only S z c is important. 

As expected, the sensitivity is the best for the sensor with the highest pixel resolution. 
Sensitivity also improves as the sensor is moved closer to the ground. It becomes poor tor the 
features that are located at the far end of the vertical axis (top of the sensor), i.e., for the objects that 
are located at the far end of the runway. Thus, as expected, the position and velocity ot the aircratt 
can be computed to a better accuracy by knowing the position of stationary objects on the ground 
that are closer to the aircraft. 

The results indicate that the accuracy of camera state estimation would be no better than the 
GPS data unless a high resolution sensor is employed. Note that these results do not consider 
potential improvements that can be obtained by motion stereo techniques using a large number ol 
image frames. We are presently investigating the possibility ot improving the accuracy ol the 
computed sensor positional parameters by extending our analysis using this method. 


Sensor Characteristic 

Sensor npe Pixel 

(H x V) 

Field of View 
(H x V) deg. 

HDTV 

1920 x 1035 

30 x 24 

FLIR 

512 x 512 

28 x 21 

MMW 

80 x 64 

27 x 22 

Sensor Positions 


Location 

Range in ft : 

Altitude in ft. 


Threshold 

0.0 

50.0 

CAT II - DH 

90S. 1 

100.0 

CAT I -DH 

2S16.2 

2(X).() 

Middle Marker 

4500.0 

288.2 

1000' Altitude 

18081.1 

1000.0 

( )uter Marker 

29040.0 

1574.3 

In all the above six cases 


Pitch angle 

-3.0 degree 


Roll angle 

0.0 degree 


Cross Rane 

e 0.0 ft. 



Table II. 


Sensitivity in direction of Rcnqe 
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Fig. 8. 
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Fig. 9. 


Fig. 10. 


IV. Model Transformation 

As noted earlier, the PMMW images are low contrast-low resolution images. Simple edge 
detection techniques on these images generate man' : oisy edge pixels in addition to those 
belonging to the true edges such as runways, sky etc. This problem is alleviated by defining regions 
where the true edges are expected to occur using knowledge about the aircraft position and a model 
ot the airport. The main I unctions ot the model transformation module is to define a region of 
interest on the ground plane lor each leature in the model and to perform 3D to 2D transformation. It 
also defines a region in the image plane where the horizon line should occur. 


A. Defining Regions of Interest for Runway Edges 

The error in the expected location ot a feature and its actual position in the image depends on 
several factors, most notably the accuracy ot the camera position parameters used by the model 
transformation module. Furthermore, it is evident from our earlier analysis (Fig. 6) that the ground 
area covered by a pixel is a function ot the position of the pixel in the image. Thus it is not 
reasonable to define the search space tor each feature as a fixed number of pixels centered around 
the expected location in the image plane. Hence we define the region of interest in the 3D space and 
then apply transformation to get the corresponding region ot interest in the image. The extent of the 
search space in the 3D space is determined by the estimated error in camera positional parameters 
(which are based on GPS and on-board instrument data). 

The geometric model of the airport contains a sequence of 3D coordinates of the vertices of 
the runwav/taxiwavs, which forms a polygon with n vertices: 

runway = {/>■}, /=/, 2 n , 
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where P, - <Xj, Z,j^ is one of the vertices of the polygon. Note that Z, - 0. PjPj + 1 specifies an 
edee of the polygon. The region of interest is detined as a rectangle on the ground which encloses 
the edge. Therefore, each edge P,P, +l of the polygon is associated with the region of interest 

defined by four points b, = (Xy,Ty,Zj) , j=l 4, and Zj = 0. 

The width of the region of interest is defined as a function of the width oi the runway/taxiway, 
tv. accuracy of the GPS data, g ( g < 1), and the accuracy of the on-board instruments, d (d< 1). 

Note that g and d are determined by the specification and characteristics of these instruments. This 
relationship is given by 

0.2vi ,f.) 

width(w.g,d) = — • ■ 

Note that the minimum width is 0.2w when g—d—1, which corresponds to ±10$ potential 
displacement of runway edge feature. To limit the search area from being a large fraction of the 

runway width we limit the search width to <).4w even il gd<0.5. 

After defining the region of interest lor each edge. 3D to 2D coordinate transtormation is 

performed using the following homogeneous equation |3]: 
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are the perspective projection, rotation and translation transtormation matrices, respectively, and /is 
the focal length. After perspective projection, we need to consider the tollowing special cases. 

A. the region of interest degenerates to a line in the image plane because the region is too tar 
from the camera. 
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B. the region of interest in the image plane becomes very large because the edge is very 
close to the camera. 

For case A. a minimum width in the image plane is assigned in order to provide some search space 
for the feature detector. For case B, a maximum width in image space is defined to further restrict 
the region. In our experiment, for the aforementioned extreme cases, the minimum and maximum 
width of a region of interest are set to be 10 and 20 pixels, respectively. 

B. Defining Search Space for Horizon Line 

When the vertical angular field of view is larger than 20 , then a horizon line appears in the 
image (Fig. 1 1 ). The horizon is an important clue in estimating the camera orientation since it gives 
the roll angle information directly. Search space in the image plane is defined to locate this line. 




Fig. 1 1. Horizon line in the image. 


Without loss of generality, consider the situation when the aircraft is heading towards the X 
axis of the world coordinate system. Assume the camera is located at point D (see Fig. 11) with 
pitch angle 6, and zero yaw and roll angles. Points A and B are on the top and bottom edge of the 
image, respectively. The horizon will then appear horizontally in the image plane as shown. The 

distance between this line and the center line of the image is given by HC = /tan(0). Since in the 
above analysis roll angle has been assumed to be zero, the horizon appears parallel to the horizontal 
axis of the image plane. For any non zero roll angle, a simple roll transformation on this line will give 
the horizon in the image. The associated region of interest is defined to be 10 pixels centered around 
the expected horizontal line in the image. 

It is possible for the projection of the region of interest onto the image plane to be partially 
outside the image boundary. In such cases, we need to clip these regions so that the search space 
always remains within the confines of the image. This is done using the "polygon clip and fill'’ 
algorithm (4], The regions of interest for both the runway and the horizon of the image sequence 
used in these experiment are shown in Fig. 12. 
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Fig. 12. Regions of interest. 


V. Runway Localization and Object Detection 

A. Runway Localization 

In this part of the system we search for the expected features within the region ot interest, 
defined by the previous module. This will significantly reduce the search time and also avoid the 
spurious response which is likely in such a low resolution input image. An accurate localization ol 
the feature is necessary for estimation of motion parameters and camera pose. 

A Sobel edge detector is applied to the sensor image. We then select one ot the tour 
scanning directions (-45°. 0°. 45°, 90°) which is approximately orthogonal to the direction ot the 
expected edge. Along each scan line we locate pixels with greatest edge strength. As the runway 
edge is supposed to be a straight line we fit a best line to these pixels. We also associate a 
measure of confidence for these detected edges based on the number of edge pixels detected along 
the line. 

B. Object Detection 

In this section, the region inside and outside the runway/taxiways are separately checked lor 
the existence of any stationary or moving objects. The image has three homogeneous regions, 
namely the sky, the runway/taxiways and the region outside the runway/taxiways. Any objects on 
or outside the runway/taxiways are expected to have some deviation in graylevel from their 
respective homogeneous background. Hence, we use histogram-based thresholding lor object 
detection. The thresholds which determine this deviation are set to be different tor different regions. 

We generate a mask image which represents three homogeneous regions. Using this mask 
image, we generate the histogram and compute its standard deviation for each region separately 
(except for the sky region). The threshold value is determined as a function ot the mean and the 
standard deviation, and any area which has graylevel lower than the threshold is considered as 
object regions. An object is assumed to have a reasonable size. This size restriction on the object 
can be used to ignore spurious responses resulting from the thresholding. Each object is then 
labeled based on 4-connectivity. 
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VI. Experimental Results 

We have tested our algorithm on a test image provided by the TRW. This image was 
obtained using a single pixel camera located at a fixed point in space (a camera with an array of 
pixels is under development). The camera was mechanically scanned to obtain a 50X150 pixel 
image. This is the image shown in Fig. 2. We were also provided with the model of the runway 
giving the 3D world coordinates of the runway comers, locations of the buildings etc. Using these 
data and the single image, we created a sequence of 30 frames to simulate the images from a moving 
camera. Frames 1 (original), 5, 10, and 15 from this sequence is shown in Fig. 13(a). Edge 
enhanced images corresponding to these frames are shown in Fig. 13(b). The regions of interest 
defined on these frames are shown in Fig. 13(c). Delineated features superimposed on the images 
are shown in Fig. 13(d). Although all the edges are detected accurately in this example, it is likely 
that one or more edges of a polygon are not detected. To handle such situations we associate a 
degree of importance for each edge. For example, runway edges which are closer to the camera must 
be detected in the image whereas those corresponding to the far end of the runway are usually very 
short and may or may not be detected. And overall confidence measure is associated with each 
detected region. 

Objects detected on the runway in Frame 1 and those outside the runway are shown in 
Fig. 14. Warning signals are generated for each object on or near runway. Algorithms to track these 
in successive frames and estimate camera state using motion stereo are under development. 



Frame 1 


Frame 5 



Frame 10 


Frame 15 


(a) 

Fig. 13. The input images (a), edge images (b), regions of interest (c), and 
detected features superimposed on the original images (d). 
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Fig. 14. Detected objects inside (left) and outside (right) the runway. 


VII. Future Work and Conclusions 

In this paper, we have described a vision-based system to assist pilots during landing under 
restricted visibility conditions. The images obtained by a passive sensor is processed to detect 
major regions such as runways and objects inside and outside these regions. The image resolution 
is very poor; however, additional information in the form of airport geometric model, and camera 
position parameters are available to guide the segmentation algorithms. Objects are detected in each 
of these regions using thresholds computed separately for each region. Our results show that the 
model-based feature detection approach is quite accurate and the homogeneity assumption on 
regions for object detection is reasonable. The success of this model-based approach clearly 
depends upon the accuracy of the camera position parameters used to define search regions in the 
image. One of the methods for updating camera position information is triangulation using known 
objects. We have derived the accuracy of such an update as a function of camera characteristics and 
image parameters. 

At this stage, our system is able to detect the runway/taxiways and the objects inside and 
outside the runway/taxiways in each frame and to report their positions in the image. Since we have 
a moving camera, moving object situation, even the stationary objects appear to be moving in the 
image. Work is in progress to estimate the egomotion of the camera, to distinguish moving objects 
from stationary ones and to estimate the velocities of the moving objects. There is also potential to 
obtain more accurate camera state estimation using motion stereo from image sequences compared 
to using GPS data alone. 
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